Support pandas 3+ in the dataframes API (#40709) - #40768

📋 Review Summary

This PR updates the DataFrame API, schema conversion utilities, and type compatibility test suites to support Pandas 3.0 (where infer_string is enabled by default) and adds py311-pandas-30 to tox.ini. While the core Pandas 3.0 compatibility fixes are solid, the setup.py package bounds and Gradle test tasks are not updated yet, and one schema conversion path still relies on dtype.kind == 'O' instead of pd.api.types.is_string_dtype.

🔍 General Feedback

• Dependency & CI Wiring (setup.py & build.gradle): sdks/python/setup.py still caps dataframe_dependency at 'pandas>=2.2.2,<2.4' (with # TODO(https://github.com/apache/beam/issues/40709): support pandas 3+) and docs/test extras at 'pandas<2.6.0'. Additionally, sdks/python/test-suites/tox/py311/build.gradle needs a testPy311pandas-30 task registered so the new py311-pandas-30 tox environment is actually executed in postCommitPyDep (and .github/trigger_files/beam_PostCommit_Python_Dependency.json can be bumped to trigger it on the PR).
• Remaining dtype.kind == 'O' check in schemas.py: sdks/python/apache_beam/dataframe/schemas.py (line 206 in element_type_from_dataframe) still checks if dtype.kind == 'O': return Any when inferring schema types from a proxy without element_type. Under Pandas 3.0, default string columns have pd.StringDtype(na_value=np.nan) (dtype.kind == 'U'), which bypasses that branch and falls through to dtype_to_fieldtype (raising TypeError: Unable to encounter corresponding Beam type for dtype=str). Updating that check to if dtype.kind == 'O' or pd.api.types.is_string_dtype(dtype): (or handling pd.api.types.is_string_dtype in pandas_type_compatibility.py) will keep string column fallback consistent.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

💬 Inline Findings

• sdks/python/tox.ini (line 376): 🟠 High: py311-pandas-30 is added here in tox.ini, but two other places need to be updated in tandem:

1. sdks/python/test-suites/tox/py311/build.gradle needs a corresponding toxTask "testPy311pandas-30", "py311-pandas-30", "${posargs}" (added to test and postCommitPyDep) so that CI actually runs this tox environment.
2. sdks/python/setup.py still restricts dataframe_dependency to 'pandas>=2.2.2,<2.4' (and docs/test extras to 'pandas<2.6.0'). Since tox installs the package with extras = test,dataframe,..., pip will either report a version conflict in commands_pre (pip check) or prevent users from installing Pandas 3.0 with apache-beam[dataframe].

• sdks/python/apache_beam/dataframe/schemas_test.py (line 227): 🟡 Medium: When major >= 3, series_tests is reassigned instead of appended to (+=), which drops all the numeric and nullable integer Series test cases defined in lines 211–218 on Pandas 3+.

      series_tests += [(pd.Series(['a', 'b', 'c'], dtype=dtype), str)]

This changes the pandas version for every Python 3.11+ user of the dataframe extra and for every Python 3.11+ precommit. Through the weekly dependency update, it also changes the default containers, which carry users' own pandas code too. That's a large blast radius, so I think we should scope it out before raising the cap, as #40709 asks.

Here is a potential approach:

1. List what changed in pandas. Go through the 3.0 and 3.1 release notes (breaking changes, removed deprecations, changed defaults). Write down each item that could reach Beam.
2. List where Beam depends on pandas. That means:
   • the DataFrame API, which wraps pandas methods by name and signature;
   • conversion between dtypes and Beam schemas;
   • DataFrame IO;
   • other modules that use pandas (CSV IO, YAML, interactive, ML handlers);
   • the default containers.
3. Match the two lists up and decide the behaviour. For each change that reaches Beam, decide whether Beam follows pandas 3, keeps the old behaviour, or marks it unsupported. The changes that alter results without raising an error need an explicit decision, not just a fix. Examples are the new string dtype, copy-on-write, and changed defaults.
4. Start on pandas 2.3. pandas 2.3 can switch on the pandas 3 string dtype (future.infer_string) and copy-on-write (mode.copy_on_write), and it warns about features pandas 3 removes. Run Beam's tests with those switched on and warnings treated as errors. That finds many problems while the cap stays where it is.
5. Test every version the cap allows. Run the full tests, including the pandas doctest suite, on both ends of the supported 2.x range and on each 3.x minor version. Group the failures by cause rather than by test.
6. Look for what the tests miss.
   • Compare Beam's results with pandas' on the same inputs, across argument combinations, not just the defaults.
   • Use a script to compare the signatures of wrapped methods between pandas versions.
   • Run each dtype through a real pipeline (to_dataframe → to_pcollection) and check the output types and missing values.
   • Check that every version-specific branch actually runs on that version.
7. Check rollout cases.
   • Launcher and workers on different pandas versions.
   • Pipeline update compatibility where output schemas change.
   • What the default containers ship, and when.

The result could be a short write-up on #40709 listing:

• each breaking change and the decision for it;
• the remaining work;
• the rollout plan.

A low-risk rollout would be:

1. Land the fixes with the cap still below 3, checking each on pandas 2.x.
2. Add a CI env for each 3.x version once it passes.
3. Raise the cap in a separate PR with a CHANGES.md entry.